[SC-17128] ZD-717: Add multiclass (one-vs-rest) support to VM tests#535
Conversation
Extend previously binary-only sklearn model-validation tests to support multiclass targets, computed one-vs-rest with a micro-average where applicable and sourced from the underlying estimator's per-class predict_proba (skipped gracefully when unavailable): - ROCCurve: per-class OvR ROC curves + micro-average; docstring updated - PrecisionRecallCurve: per-class OvR precision-recall curves - ConfusionMatrix: multiclass uses argmax predictions (threshold ignored) - PopulationStabilityIndex: one-vs-rest per-class PSI - Unit tests for the ROC and PR multiclass paths - Add multiclass_classification_tests notebook code sample SC-17128 / ZD-717 Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
cachafla
left a comment
There was a problem hiding this comment.
Tested the notebook and is working great 👏
If we're going to distribute this notebook I suggest:
- Ensuring it has the same structure as other public notebooks (intro text, code snippet, footer, etc.
- Put it inside a directory, right now it's at the root of
code_samples/
Otherwise, we can keep it inside code_sharing/.
It is not something for demonstration. We can keep in inside |
PR SummaryThis PR introduces comprehensive enhancements and bug fixes to the model validation tests for sklearn classification models. The key changes focus on adding explicit multiclass handling for several metrics, including ROC Curve, Precision-Recall Curve, Confusion Matrix, and Population Stability Index. The updates include:
Overall, this PR refines the behavior for classification model validations by accommodating multiclass support, ensuring graceful degradation for incompatible models, and confirming the outputs with robust unit tests. Test Suggestions
|
…on-multiclass-models-follow-up-to [SC-17187] ZD-717: GINITable fails on multiclass models (follow-up to PR #535)
…(ZD-717) Follow-ups from the PR #539 review. ROCCurve, PrecisionRecallCurve, PopulationStabilityIndex (#535) and GINITable (#539) each carried an identical ~40-line preamble to fetch the underlying estimator's per-class probability matrix for their one-vs-rest multiclass paths. Extract it into sklearn/_multiclass_proba.py (same pattern as _diagnosis_metrics.py) and point all four tests at it. Fix the class/column alignment while there: the preamble assumed np.unique(dataset.y) matches predict_proba column order, but sklearn orders columns by estimator.classes_ (training classes). Align on classes_ with a np.unique fallback, binarize against the full training class list, and emit per-class output only for classes present in the evaluated y. Net behavior change: an evaluation slice missing a training class now computes metrics for the present classes plus micro-average instead of raising SkipTestError. Two guards the old width check provided implicitly are kept explicit: a binary-trained model evaluated on a >2-class dataset skips (previously the new alignment would have crashed with IndexError), and evaluation labels the model was never trained on skip rather than being silently absorbed as all-negative rows. Adds unit tests for the helper (10) and un-gated sklearn multiclass tests for all four call sites, including the missing-class scenario; the pre-existing multiclass tests for ROC/PR were entirely xgboost-gated, so the refactored paths now have coverage without the extra. PSI gains its first unit test file. 114 passed / 5 xgboost-gated skips across tests/unit_tests/model_validation. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Pull Request Description
What and why?
Adds multiclass (one-vs-rest) support to four sklearn model-validation tests that were
previously binary-only, so customers validating multiclass classifiers get usable results
instead of a skipped/binary-only test (requested via ZD-717).
Per-class probabilities are read from the underlying estimator's
predict_proba— theVMModelwrapper only exposes the positive-class column, which is binary-only — and the truelabels are one-hot encoded with
label_binarizein sklearn's sorted-class order to stayaligned with the
predict_probacolumns.with its own AUC;
RawDatareturns per-classfpr/tpr/auckeyed by class label plus amicroentry. Docstring updated for multiclass.thresholdparameter is binary-only and is ignored for multiclass.Binary behavior is unchanged in all four. Where a full per-class probability matrix isn't
available (metadata-only models, precomputed single-column probabilities, or a shape that
doesn't match the class count), the multiclass path raises a clear
SkipTestErrorrather thancrashing.
How to test
Automated tests
Run the added/updated unit tests (this repo runs unittest via
uv):New multiclass coverage:
TestROCCurveMulticlass.test_multiclass_one_vs_rest_tracesandTestPrecisionRecallCurveMulticlass.test_multiclass_one_vs_rest_traces(both train a 3-classXGBoost model and assert one curve per class + a micro-average trace + baseline, and per-class
AUCs above 0.5). Binary tests in the same files should continue to pass.
Manual testing
Run
notebooks/code_samples/multiclass_classification_tests.ipynbend-to-end against a3-class model and confirm:
What needs special review?
label_binarize(y, classes=...)must matchpredict_probacolumn order (sklearn's sorted class labels). Misalignment would silently mislabel per-class
curves.
SkipTestErrorfallbacks should skip only genuinely unsupportedmodels (no per-class
predict_proba, wrong-shaped probability matrix) and not falsely skipsupported ones.
thresholdis now ignored for multiclass — confirm this is the desiredbehavior and is clearly documented.
Dependencies, breaking changes, and deployment notes
None. Library-only change, no migrations, no new env vars, no companion frontend/backend PR.
Binary output is unchanged, so no breaking changes.
Shortcut: SC-17128 ·
Release notes
ROC, Precision-Recall, Confusion Matrix, and Population Stability Index tests now support
multiclass classification models, in addition to binary. For models with more than two
classes, ROC and Precision-Recall curves are drawn one-vs-rest (one curve per class plus a
micro-average), the confusion matrix uses the model's predicted class, and PSI is reported per
class.
Checklist
enhancementtest_ROCCurve+test_PrecisionRecallCurvepass (5 tests, incl. multiclass OvR)